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Highlights 

Highlights  of  GAO-08-982,  a  report  to 
congressional  committees 


Why  GAO  Did  This  Study 

Since  the  September  2001  terrorist 
attacks,  Congress  has  provided 
about  $800  billion  as  of  July  2008  to 
the  Department  of  Defense  (DOD) 
for  military  operations  in  support 
of  the  Global  War  on  Terrorism 
(GWOT).  GWOT  budget  requests 
have  grown  in  scope  and  the 
amount  requested  has  increased 
every  year.  DOD  uses  various 
processes  and  the  Contingency 
Operations  Support  Tool  (COST)  to 
estimate  costs  for  these  operations 
and  to  develop  budget  requests. 

GAO  assessed  (1)  how  DOD  uses 
COST  and  other  processes  to 
develop  GWOT  budget  requests 
and  (2)  what  actions  DOD  has 
taken  to  ensure  COST  adheres  to 
best  practices  for  cost  estimation. 

GAO  interviewed  DOD  officials  and 
others  to  determine  how  the 
services  develop  GWOT  budget 
requests  using  COST  and  other 
processes.  GAO  also  used  its  Cost 
Assessment  Guide  as  criteria  for 
best  practices  for  cost  estimation. 


What  GAO  Recommends 


GAO  is  recommending  that  DOD 
(1)  arrange  for  an  independent 
review  of  COST  to  ensure  that  the 
model  adheres  to  best  practices 
and  (2)  consider  options  for 
refining  COST  to  better  meet  the 
needs  of  the  services.  DOD  agreed 
with  both  of  GAO’s 
recommendations. 


To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-982. 

For  more  information,  contact  Sharon  L. 
Pickup,  (202)  512-9619,  pickups@gao.gov. 


DEFENSE  BUDGET 

Independent  Review  Is  Needed  to  Ensure  DOD's  Use 
of  Cost  Estimating  Tool  for  Contingency  Operations 
Follows  Best  Practices 


What  GAO  Found 

The  services  use  COST  as  part  of  their  process  to  develop  a  GWOT  budget 
request.  While  the  Army  relies  more  on  the  estimate  resulting  from  COST,  the 
other  services  adjust  the  results  of  COST  to  reflect  estimates  they  generate 
outside  of  COST,  based  on  historical  obligation  data  and  other  information. 
DOD’s  financial  management  regulation  and  other  guidance  require 
components  to  use  COST  to  develop  an  estimate  for  the  deployment  and 
sustainment  of  military  personnel  and  equipment  for  ongoing  operations  in 
support  of  GWOT.  While  all  services  use  COST  to  develop  an  initial  estimate. 
Air  Force,  Marine  Corps,  and  Navy  budget  officials  alter  the  results  of  the  tool 
to  match  information  provided  by  lower  level  commands  and  historical 
obligation  data  that  they  believe  are  more  accmate  than  the  COST-generated 
estimate.  These  officials  stated  that  the  tool  routinely  overestimates  some 
costs  and  therefore  most  changes  made  are  decreases  in  the  amount 
estimated  by  COST.  These  officials  believe  that  the  requirement  to  use  COST 
to  develop  a  GWOT  budget  request  is  a  duplicative  process  to  their  preferred 
method  of  using  historical  obligation  data  and  other  information  better  suited 
to  their  specific  service.  For  example,  they  stated  that  COST  better  represents 
the  needs  of  Army  groimd  forces  and  the  tool  has  not  been  refined  to  be  as 
effective  for  estimating  needs  for  their  service’s  mission.  These  officials  also 
mentioned  that  COST  is  better  suited  for  developing  estimates  for  smaller- 
scale  contingency  operations  than  for  the  lengthy  deployments  and 
sustainment  phases  associated  with  a  large  campaign  such  as  GWOT.  To 
develop  estimates  for  items  that  are  outside  the  scope  of  COST,  such  as 
procurement  and  certain  contracts,  the  military  services  rely  primarily  on 
needs  assessments  developed  by  commanders  and  historical  obligation  data. 

DOD  has  taken  steps  to  improve  the  performance  and  reliability  of  COST; 
however,  COST  could  benefit  from  an  independent  review  of  the  tool’s 
adherence  to  best  practices  for  high-quality  cost  estimation  as  described  in 
GAO’s  Cost  Assessment  Guide.  COST  has  been  refined  many  times  and  cost 
factors  are  routinely  updated  in  an  effort  to  use  the  most  current  information 
available  to  develop  an  estimate.  DOD  officials  stated  they  are  confident  in  the 
tool’s  ability  to  provide  reasonable  estimates  because  COST  is  frequently 
updated.  However,  COST  has  not  been  assessed  against  best  practices  for  cost 
estimation  to  determine  whether  COST  can  provide  high-quality  estimates  that 
are  well  documented,  comprehensive,  accurate,  and  credible.  While  GAO  did 
not  undertake  a  full  assessment  of  COST  against  best  practices,  it  determined 
that  some  features  of  the  tool  meet  best  practices  while  other  features  would 
benefit  from  further  review.  For  example,  the  tool  adheres  to  several  best 
practices  for  a  comprehensive  and  accurate  cost  estimate,  such  as  frequent 
updates  to  the  structure  of  COST  and  the  data  the  tool  uses  to  generate 
estimates.  However,  COST  relies  on  GWOT  obligation  data  that  GAO  has 
identified  as  being  of  questionable  reliability.  A  thorough,  independent  review 
of  COST  against  best  practices  could  provide  decision  makers  with 
information  about  whether  the  tool  creates  cost  estimates  for  GWOT 
expenses  that  are  well  documented,  comprehensive,  accurate,  and  credible. 
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United  States  Government  Acconntability  Office 
Washington,  DC  20548 


September  16,  2008 
Congressional  Committees 

Between  the  terrorist  attacks  of  September  11,  2001,  and  July  2008, 
Congress  has  provided  about  $800  billion  to  the  Department  of  Defense 
(DOD)  for  domestic  and  overseas  military  operations  in  support  of  the 
Global  War  on  Terrorism  (GWOT).^  Over  the  years,  DOD  has  used  different 
approaches  to  request  GWOT  fimding,  including  yearly  emergency 
supplemental  budget  requests,  amendments  to  prior  supplemental 
requests,  and  requests  included  as  part  of  the  President’s  annual  budget 
submission.  GWOT  budget  requests  have  increcised  from  $14  billion  for 
fiscal  year  2002  to  $189  billion  for  the  total  fiscal  year  2008  request.  The 
DOD  components  responsible  for  carrying  out  activities  in  support  of 
GWOT  use  various  processes  to  develop  budget  requests  for  GWOT. 

DOD’s  financial  management  regulation  for  contingency  operations^  and 
other  guidance  require  military  components  to  use  a  DOD-sponsored  cost 
estimation  tool,  called  the  Contingency  Operations  Support  Tool  (COST), 
to  develop  cost  estimates  for  the  deployment  and  sustainment  of  military 
personnel  and  equipment  for  GWOT.  This  tool  was  constructed  10  years 
ago  to  develop  estimates  for  smaller-scale  contingency  operations,  such  as 
peacekeeping  operations  and  disaster  relief  operations,  but  is  now  being 
used  to  also  develop  a  portion  of  cost  estimates  for  the  large-scale  GWOT 
operations  in  Iraq  and  Afghanistan. 

While  COST  has  the  capability  to  develop  an  estimate  for  military 
personnel  and  operating  costs,  it  does  not  have  the  capability  to  estimate 
other  costs  such  as  procmement,  some  reset-leveF  maintenance  of 


^After  the  terrorist  attacks  of  September  11,  2001,  the  President  announced  a  global  war  on 
terrorism  requiring  the  collective  instruments  of  the  entire  federal  government  to  counter 
the  threat  of  terrorism.  Ongoing  military  and  diplomatic  operations  overseas,  especially  in 
Iraq  and  Afghanistan,  constitute  a  key  part  of  GWOT.  These  operations  involve  a  wide 
variety  of  activities,  such  as  combating  insurgents,  civil  affairs,  capacity  building, 
infrastructure  reconstruction,  and  training  military  forces  of  other  nations. 

^DOD,  Financial  Management  Regulation,  7000.14-R,  vol.  12,  ch.  23  (September  2007). 

A  January  2007  DOD  memorandum  defines  reset  as  actions  to  restore  units  to  a  desired 
level  of  combat  capability,  including  maintenance  and  supply  activities  that  restore  and 
enh2mce  equipment  that  was  destroyed,  damaged,  stressed,  or  worn  out  beyond  economic 
repair  due  to  combat  operations  by  repairing,  rebuilding,  or  procuring  replacement 
equipment. 
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equipment,  and  certain  contract  costs.  For  the  first  few  years  of  GWOT, 
COST-generated  estimates  for  personnel  and  operating  costs  ranged  from 
around  80  to  50  percent  of  the  total  budget  request  for  GWOT  for  fiscal 
years  2003  to  2006,  respectively.  However,  this  COST-generated  portion  of 
the  total  has  continued  to  decrease  every  year.  For  the  fiscal  year  2008 
request  submitted  in  February  2007,  the  COST-generated  portion  of  the 
request  was  about  30  percent  due  to  a  significant  increase  in  certain  types 
of  estimated  costs,  such  as  procurement  and  reset-related  maintenance, 
which  are  outside  the  scope  of  COST.  Estimates  for  these  types  of  costs, 
or  the  remaining  70  percent  of  the  fiscal  year  2008  request,  are  developed 
using  other  models  and  formulas,  historical  obligation  data,  and  requests 
submitted  by  commanders.  Each  component  submits  its  estimate  for  its 
GWOT  budget  request,  including  the  COST-related  portion  for  military 
personnel  and  operations  needs  and  the  portions  developed  by  other 
means,  to  the  Office  of  the  Under  Secretary  of  Defense  (Comptroller).^ 
These  estimates  are  reviewed  and  revised  by  the  DOD  Comptroller  and  the 
Office  of  Management  and  Budget  before  they  are  submitted  to  Congress. 

Because  federal  guidelines  are  limited  on  the  processes,  procedures,  and 
practices  for  ensuring  credible  cost  estimates,  we  reviewed  existing  best 
practices  and  developed  the  Cost  Assessment  Guide  to  fill  that  gap.*^  The 
best  practices  identified  in  our  guide  were  compiled  from  a  thorough 
review  of  relevant  legislation,  regulations,  policy,  and  guidance  for  the 
criteria  that  most  pertained  to  cost  estimating.  Our  guide  defines  cost 
estimation  as  the  process  of  using  estabhshed  methods  and  valid  data  to 
estimate  the  future  costs  of  a  program  based  on  what  is  known  today,  such 
as  the  use  of  COST  to  develop  a  portion  of  GWOT  budget  requests.  DOD 
and  the  Office  of  Management  and  Budget  have  issued  the  majority  of 
guidance  and  regulations  regarding  cost  estimating,  and  the  best  practices 
in  our  guide  were  compiled  from  these  and  other  sources.  The 
characteristics  of  high-quality  cost  estimation  support  a  process  that  is 
well  documented,  comprehensive,  accurate,  and  credible. 

Over  the  past  several  years,  we  have  conducted  a  series  of  reviews  under 
the  authority  of  the  Comptroller  General  examining  the  funding  and 
reported  obligations  for  operations  in  support  of  GWOT.  We  have 


^For  the  purpose  of  this  report,  we  refer  to  the  Office  of  the  Under  Secretary  of  Defense 
(Comptroller)  as  the  DOD  Comptroller. 

^GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estimating  and  Managing  Program 
Costs,  GAO-07-1134SP  (Washington,  D.C.:  July  2007). 
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previously  reported  that  there  are  reliability  concerns  regarding  DOD’s 
reported  GWOT  obligations,  and  problems  exist  with  transparency  over 
certain  costs.  We  have  made  a  series  of  recommendations  to  the  Secretary 
of  Defense  intended  to  improve  the  transparency  and  reliability  of  DOD’s 
reporting  of  GWOT  obligations  and  to  adjust  GWOT  funding  requests.® 
DOD  has  implemented  many  of  these  prior  recommendations  and 
continues  to  improve  its  cost  reporting  efforts.  Because  of  broad 
congressional  interest  in  issues  related  to  GWOT,  we  prepared  this  report 
imder  the  Comptroller  General’s  authority  to  conduct  evaluations  on  his 
own  initiative.  Due  to  increasing  budget  requests  for  DOD’s  ongoing 
operations  in  support  of  GWOT  and  to  assist  members  of  Congress  in  their 
oversight  role  as  they  consider  future  GWOT  funding,  we  evaluated  DOD’s 
efforts  to  develop  its  GWOT  budget  requests.  Specifically,  we  assessed 
(1)  how  DOD  uses  COST  and  other  processes  to  develop  GWOT  budget 
requests  and  (2)  what  actions  DOD  has  taken  to  ensure  COST  adheres  to 
best  practices  for  cost  estimation. 

To  examine  how  DOD  uses  COST  and  other  processes  to  develop  budget 
requests  for  GWOT,  we  met  with  key  officials  from  the  DOD  Comptroller, 
the  Joint  Chiefs  of  Staff,  and  various  service  officials  from  the  Air  Force, 
Army,  Marine  Corps,  and  Navy  to  understand  their  use  of  COST  to  develop 
estimates  for  the  deployment  and  sustainment  of  troops  and  equipment 
and  about  other  processes  used  to  develop  portions  of  GWOT  budget 
requests  that  are  outside  the  scope  of  COST,  such  as  procurement  and 
certain  types  of  contracts.  We  also  obtained  their  perspectives  on  their 
experiences  using  COST  and  other  processes  to  develop  a  cost  estimate 
for  yearly  emergency  GWOT  budget  requests  or  amendments  to  those 
requests.  We  interviewed  representatives  from  the  Institute  for  Defense 
Analyses  (IDA) — the  contractor  that  developed  and  maintains  COST — ^to 
understand  the  development,  function,  and  role  of  COST.^  We  reviewed 
DOD  guidance  and  regulations  as  well  as  COST-related  documents  such  as 
briefings  and  the  COST  training  manual.  We  also  analyzed  an  example  of 
how  the  cost  estimate  for  the  operations  portion  of  an  amendment  to  a 
yearly  GWOT  supplemental  request  was  developed.  To  assess  the  actions 
DOD  has  taken  to  ensure  COST  adheres  to  best  practices  for  cost 


®GAO,  Global  War  on  Terrorism:  DOD  Needs  to  Take  Action  to  Encourage  Fiscal 
Discipline  and  Optimize  the  Use  of  Tools  Intended  to  Improve  GWOT  Cost  Reporting, 
GAO-08-68  (Washington,  D.C.:  Nov.  6,  2007). 

^The  Institute  for  Defense  Analyses  is  a  federally  funded  research  and  development  center 
that  developed  and  maintains  COST  under  a  contract  with  DOD. 
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estimation,  we  reviewed  COST-related  documents,  such  as  the  manual  for 
COST  use  and  financial  management  regulations.  Our  Cost  Assessment 
Guide  provides  best  practices  for  cost  estimating  processes.  We 
interviewed  DOD  and  service  officials  and  representatives  from  IDA  about 
steps  that  have  been  taken  to  improve  GOST’s  effectiveness  and 
functionality.  Finally,  we  analyzed  documents  and  testimonial  evidence 
from  DOD  officials  and  IDA  representatives  and  compared  this 
information  to  the  four  characteristics  of  high-quality  cost  estimation. 
While  we  did  not  conduct  a  full  assessment  of  COST  against  best 
practices,  we  identified  some  features  of  the  tool  that  either  met  best 
practices  or  appeared  to  not  meet  best  practices. 

We  conducted  this  performance  audit  from  July  2007  through 
September  2008  in  accordance  with  generally  accepted  government 
auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 


Results  In  Brief 


While  the  military  services  use  COST  as  part  of  their  process  to  develop  a 
GWOT  budget  request,  the  Army  relies  more  on  the  estimate  resulting 
from  COST  than  the  other  services,  which  significantly  adjust  the  results 
of  COST  to  reflect  estimates  they  generate  outside  of  COST.  These 
estimates  are  based  on  historical  obligation  data  and  other  information, 
which  they  believe  is  more  representative  of  future  GWOT  budget  needs. 
DOD’s  financial  management  regulation  and  other  guidance  require  the 
components  to  use  COST  to  develop  an  estimate  for  the  deployment  and 
sustainment  of  military  personnel  and  equipment  for  ongoing  operations  in 
support  of  GWOT,  and  the  guidance  additionally  provides  certain 
parameters  and  assumptions  to  be  used  in  the  tool.  While  all  the  services 
use  COST  to  develop  an  initial  estimate.  Air  Force,  Marine  Corps,  and 
Navy  budget  officials  stated  that  they  alter  the  results  of  the  tool  to  match 
information  provided  by  lower  level  commands  and  historical  obligation 
data  that  they  believe  are  more  accurate  than  the  COST-generated 
estimate.  These  officials  report  that  the  required  process  of  using  COST  to 
develop  cost  estimates  is  duplicative  to  their  preferred  process  of  using 
historical  obligation  data  and  other  information  better  suited  to  their 
specific  service  to  develop  a  GWOT  budget  request.  For  example,  they 
stated  that  while  COST  better  represents  Army  ground  forces  than  the 
unique  characteristics  of  the  other  forces,  the  tool  has  not  been  refined  to 
be  as  effective  for  estimating  certain  needs  for  their  service’s  mission. 


Page  4 


GAO-08-982  Defense  Budget 


These  officials  also  mentioned  that  COST  is  better  suited  for  developing 
estimates  for  small-scale,  short-duration  contingency  operations  than  for 
the  lengthy  deployments  and  sustainment  phases  associated  with  a  large 
campaign  such  as  GWOT.  For  the  remaining  types  of  funding  needs  that 
are  outside  the  scope  of  COST,  such  as  procurement  and  certain  contracts, 
DOD  guidance  or  regulations  do  not  prescribe  set  processes  for  estimating 
these  types  of  costs.  As  a  result,  all  military  services  rely  primarily  on 
needs  assessments  developed  by  commanders  and  historical  obligation 
data  to  develop  estimates  for  these  types  of  costs. 

DOD  has  taken  steps  to  improve  the  performance  and  reliability  of  COST; 
however,  COST  could  benefit  from  an  independent  review  of  the  tool’s 
adherence  to  best  practices  for  high-quality  cost  estimation.  For  example, 
COST  has  been  refined  many  times  over  the  past  several  years  and  cost 
factors  are  routinely  updated  in  an  effort  to  use  the  most  current 
information  available  to  develop  an  estimate.  However,  COST  has  not 
been  assessed  against  best  practices  for  cost  estimation  to  determine 
whether  COST  can  provide  high-quality  estimates  that  are  well 
documented,  comprehensive,  accurate,  and  credible.  While  we  did  not 
undertake  a  full  assessment  of  COST  against  best  practices,  during  the 
course  of  our  review  we  identified  some  features  of  the  tool  that  meet  best 
practices,  as  well  as  other  features  that  would  benefit  from  further  review. 
For  example,  the  tool  meets  several  best  practices  that  help  create  a 
comprehensive  and  accurate  cost  estimate,  such  as  frequent  updates  to 
the  structure  of  COST  and  the  data  that  COST  uses  to  generate  estimates. 
However,  while  COST  appears  to  largely  encompass  the  types  of  costs  that 
are  incurred  to  deploy  and  sustain  Army  ground  forces  and  their  related 
equipment,  it  does  not  comprehensively  or  accurately  estimate  some  costs 
for  the  other  services.  The  tool  also  relies  on  GWOT  obligation  data  that 
we  have  previously  identified  as  being  of  questionable  reliability.  DOD 
Comptroller  officials  stated  they  are  confident  in  the  tool’s  ability  to 
provide  reasonable  estimates  because  COST  is  frequently  updated. 
However,  COST  has  not  been  independently  verified  as  an  effective  tool 
for  the  estimation  of  GWOT  costs.  Based  on  our  Cost  Assessment  Guide, 
we  believe  that  a  thorough,  independent  review  of  COST  against  best 
practices  could  provide  decision  makers  with  information  about  whether 
the  tool  creates  a  cost  estimate  for  GWOT  expenses  that  is  well 
documented,  comprehensive,  accurate,  and  credible. 

To  ensure  that  the  DOD  budget  requests  for  GWOT  are  based  on  a  sound 
cost  estimation  process,  we  recommend  that  the  Secretary  of  Defense 
direct  the  Under  Secretary  of  Defense  (Comptroller)  to  arrange  for  an 
independent  review  of  COST.  Furthermore,  based  on  the  results  of  such  a 


Page  5 


GAO-08-982  Defense  Budget 


review  and  taking  into  consideration  how  each  service  presently  uses 
COST,  we  also  recommend  that  the  Secretary  of  Defense  direct  the  Under 
Secretary  of  Defense  (Comptroller)  to  consider  options  for  refining  COST, 
determine  the  appropriate  items  or  types  of  costs  for  which  COST  should 
be  used,  and  identify  methods  to  be  used  when  COST  is  not  appropriate.  In 
written  comments  on  a  draft  of  this  report,  DOD  agreed  with  both  of  our 
recommendations.  In  addition,  DOD  provided  technical  comments  which 
we  have  incorporated  as  appropriate.  DOD’s  comments  and  our  evaluation 
of  them  are  discussed  in  detail  in  a  later  section  of  this  report  and  the 
department’s  comments  are  reprinted  in  appendix  IV. 


Background 


Cost  estimation  is  a  difficult  process  that  requires  both  data  and  judgment, 
and  seldom,  if  ever,  are  estimates  precise — the  goal  is  to  find  a 
“reasonable”  estimate  of  future  needs.  Cost  estimates  are  necessary  for 
government  programs  for  many  reasons:  for  example,  to  support  decisions 
about  whether  to  fund  one  program  over  another,  develop  annual  budget 
requests,  or  evaluate  resource  requirements  at  key  decision  points.  As 
discussed  in  our  Cost  Assessment  Guide,  developing  a  good  cost  estimate 
requires  stable  program  requirements,  access  to  detailed  documentation 
and  historical  data,  well-trained  and  experienced  cost  analysts,  a  risk  and 
uncertainty  analysis,  and  the  identification  of  a  range  of  confidence  levels. 
The  guide  also  outlines  12  steps  for  a  high-quality  cost  estimation  process,® 
which  are  to: 

•  Define  the  estimate’s  purpose 

•  Develop  the  estimating  plan 

•  Define  the  program 

•  Determine  the  estimating  approach 

•  Identify  ground  rules  and  assumptions 

•  Obtain  the  data 

•  Develop  the  point  estimate 

•  Conduct  sensitivity  analysis 

•  Conduct  risk  and  uncertainty  analysis 

•  Document  the  estimate 

•  Present  estimate  to  management  for  approval 

•  Update  the  estimate  to  reflect  actual  costs  and  changes. 


®The  12  steps  of  the  cost  estimating  process  are  iterative  and  can  sometimes  be 
accomplished  in  varying  order  or  concurrently.  Additionally,  the  analysis,  presentation,  and 
updating  of  the  estimate  can  lead  to  repeating  previous  steps. 
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It  is  important  that  cost  estimators  and  independent  organizations  validate 
that  all  cost  elements  are  credible  and  can  be  justified  by  acceptable 
estimating  methods,  adequate  data,  and  detailed  documentation.  Hence,  in 
addition  to  the  12  steps  of  a  high-quality  cost  estimation  process,  the  guide 
also  describes  four  best  practice  characteristics  of  a  high-quality,  reliable 
estimate  generated  by  a  sound  cost  estimation  process.  Specifically,  the 
estimate  should  be  well  documented,  comprehensive,  accurate,  and 
credible.  Table  1  describes  these  characteristics  in  more  detail.  Adherence 
to  these  best  practices  can  help  ensure  that  a  cost  estimation  process 
provides  a  reasonable  estimate  of  how  much  it  will  cost  to  accomplish  all 
tasks  related  to  a  program  and  that  the  estimate  is  traceable,  accurate,  and 
reflects  realistic  assumptions. 


Table  1:  Four  Characteristics  of  a  High-Quaiity,  Reliabie  Cost  Estimate 

Characteristic 

Description 

Well  documented 

Thoroughly  documented,  including  source  data  and  significance, 
clearly  detailed  calculations  and  results,  and  explanations  of  why 
particular  methods  and  references  were  chosen.  Data  can  be 
traced  to  their  source  documents. 

Comprehensive 

Enough  detail  to  ensure  that  cost  elements  are  neither  omitted  nor 
double  counted.  All  cost-influencing  ground  rules  and  assumptions 
are  detailed  in  the  estimate’s  documentation. 

Accurate 

Unbiased,  not  overly  conservative  or  overly  optimistic,  and  based 
on  an  assessment  of  most  likely  costs.  Few,  if  any,  mathematical 
mistakes  are  present  and  those  that  are  present  are  minor. 

Credible 

Any  limitations  of  the  analysis  because  of  uncertainty  or  bias 
surrounding  data  or  assumptions  are  discussed.  Major 
assumptions  are  varied  and  other  outcomes  are  recomputed  to 
determine  how  sensitive  they  are  to  changes  in  the  assumptions. 
Risk  and  uncertainty  analysis  are  performed.  Estimate’s  results  are 
cross-checked  and  an  independent  cost  estimate  is  developed  to 
determine  whether  other  estimation  methods  produce  similar 
results. 

Source:  GAO. 


Appendix  II  presents  the  12  steps  of  a  high-quality  cost  estimation  process 
mapped  to  the  four  characteristics  of  reliable,  high-quality  estimates. 

Because  DOD’s  cost  estimates  for  military  operations  in  Bosnia  during  the 
1990s  were  consistently  well  below  the  actual  costs,  DOD  contracted  with 
IDA  to  develop  a  tool  to  assist  in  developing  preliminary  and  detailed  cost 
estimates  for  contingency  operations.  By  1998,  IDA  had  developed  the  first 
version  of  COST.  The  tool  generates  a  cost  estimate  for  a  contingency 
operation  on  the  basis  of  type  of  mission,  duration,  operational  tempo  or 
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intensity,  number  of  personnel  and  equipment,  transportation  needs, 
subsistence  for  personnel,  and  the  originating  and  destination  site. 
Accordingly,  the  tool  contains  data  relating  to  geographic  locations, 
military  unit  types,  military  equipment  types,  management  and  cost 
factors,  and  adjustment  factors  pertaining  to  climate,  terrain,  and 
operational  intensity.  Formulas  within  the  tool  draw  on  these  data  and 
user-defined  inputs,  such  as  the  number  of  personnel  or  equipment  and 
duration  of  operations,  to  develop  cost  estimates  for  many  types  of  costs 
associated  with  contingency  operations  from  predeployment  to 
reconstitution,  up  to  250  line  items  or  types  of  costs  depending  on  the 
operation  in  question.  The  tool  cannot  estimate  every  type  of  cost  that 
might  be  incurred  for  a  contingency  operation.  Rather,  it  can  only  estimate 
certain  incremental  costs  from  the  personnel,  personnel  support, 
operating,  and  transportation  cost  categories.  However,  some  types  of 
costs  within  those  categories — such  as  depot-level  maintenance  of 
equipment  and  certain  contracts,  as  well  as  all  costs  in  the  investment  cost 
category,  including  procurement  and  military  construction — are  outside 
the  scope  of  GOST’s  estimating  capabilities. 

Military  components,  including  the  services,  are  primary  sources  of  data 
for  the  tool.  Table  2  illustrates  the  primary  sources  of  data  from  each 
component  that  uses  COST  to  develop  estimates  for  GWOT  budget 
requests.’’  Cost  factors  and  management  factors  are  key  types  of  data  that 
the  tool  uses  to  develop  estimates.  Cost  factors  function  as  variables  in  the 
tool,  and  a  few  examples  are  hardship  duty  pay,  cost  for  operating  and 
support  of  a  ship,  or  average  cost  per  ton  mile  for  equipment  airlift.  Cost 
factors  are  developed  from  four  main  sources;  DOD  databases  of  record, 
service  models  for  various  statistics  such  as  Air  Force  flying  hours,  DOD’s 
Cost  of  War  reports,  and  other  information  provided  by  the  service  budget 
offices.  Management  factors  include  information  such  as  the  average 
metric  tons  per  person  of  materiel  for  deployment  or  redeployment  of  the 
monthly  flying  hours  of  aircraft. 


^For  this  review,  we  did  not  assess  the  reliability  of  any  of  these  data  sources  nor  did  we 
assess  the  appropriateness  of  these  data  sources  for  the  purposes  of  GWOT  cost 
estimation. 
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Table  2:  Primary  Sources  of  Data  within  COST  from  DOD  Components 

Military  component 

Data  source 

Army 

Army  Force  and  Organization  Cost  Estimating  System  Modei 

Army  Cost  and  Factors  Handbook 

Operating  and  Support  Management  Information  System  (part 
of  DOD’s  Visibiiity  and  Management  of  Operating  and  Support 
Costs) 

Air  Force 

Air  Force  Portai 

Air  Force  Instruction  65-503 

Navy 

Navy  Visibility  and  Management  of  Operating  and  Support 

Costs 

Navy  Budget  Office 

Marine  Corps 

1  Marine  Expeditionary  Force 

II  Marine  Expeditionary  Force 

Special  Operations 
Command  (SOCOM) 

SOCOM 

All 

Defense  Finance  and  Accounting  Service 

Defense  Heaith  Program 

Transportation  Command 

Joint  Operation  Pianning  and  Execution  System 

Source:  GAO  analysis  of  DOD  and  IDA  provided  information. 


Services  Vary  in  Their 
Reliance  on  COST  to 
Develop  GWOT 
Budget  Estimates 


The  services  use  COST  as  required  as  part  of  their  process  to  develop  a 
GWOT  budget  request,  but  the  degree  to  which  each  service  relies  on  the 
COST  results  differs.  Both  DOD  officials  and  IDA  representatives  state 
that  COST  is  better  suited  for  Army  ground  forces  and  was  primarily 
intended  to  develop  incrementaf®  personnel  and  operations  cost  estimates 
for  operations  with  discreet  phases  and  time  frames.  They  noted  it  was  not 
built  to  estimate  costs  for  the  long-term  nature  and  broad  scope  of 
activities  and  needs  related  to  operations  such  as  GWOT.  Therefore, 
service  officials  report  varying  degrees  of  confidence  in  the  tool’s 
functionality  and  accuracy  for  their  specific  service.  The  Army  relies  on 
cost’s  results  when  developing  estimates  related  to  personnel  and 
operations,  while  Air  Force,  Navy,  and  Marine  Corps  officials  often  rely 
more  on  historical  obligation  data  and  other  information  than  the 


^°The  term  “incremental  costs”  means  those  directly  attributable  costs  that  would  not  have 
been  incurred  if  it  were  not  for  the  operation. 
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COST-generated  estimates.  COST  does  not  currently  have  the  capability  to 
estimate  other  types  of  costs,  such  as  procurement,  most  equipment 
maintenance,  and  some  contract  costs.  As  a  result,  all  the  services  develop 
this  portion  of  GWOT  budget  requests  using  historical  obligation  data  and 
other  information. 


Use  of  COST  Is  Required 
asPartofDOD’sGWOT 
Budget  Request  Process 


DOD  components  are  required  to  use  COST  as  part  of  the  GWOT  budget 
request  process.  The  DOD  financial  management  regulation  that  provides 
financial  policy  and  procedures  for  small-,  medium-,  and  large-scale 
campaign  level  military  contingency  operations  requires  that  COST  be 
used  to  develop  a  cost  estimate  for  the  deployment  of  military  persormel 
and  equipment.  The  regulation  further  states  that  the  DOD  Comptroller 
will  issue  specific  guidance  providing  factors  and  cost  criteria  necessary  to 
develop  an  estimate,  and  that  the  COST  estimate  will  address  the  funding 
requirements  for  operations  and  maintenance  and  military  personnel 
costs.  The  DOD  Comptroller  issues  guidance  that  directs  the  development 
of  a  fiscal  year’s  GWOT  budget  request.  This  guidance  specifically  directs 
the  use  of  COST  to  calculate  operations  costs  related  to  GWOT,  and  also 
provides  information  guiding  COST  use,  such  as  the  level  and  intensity  of 
operations  to  be  assumed.  Much  of  the  guidance  details  the  type  and  level 
of  detail  that  must  be  provided  in  supporting  materials  that  should 
accompany  components’  estimated  GWOT  budget  requests. 


COST  does  not  develop  estimates  for  items  that  are  not  attributable  to  the 
deployment  or  sustainment  of  personnel  and  equipment,  such  as 
procurement,  most  types  of  equipment  maintenance,  and  certain  major 
contracted  needs  and  services.  As  discussed  later,  the  services  must  use 
other  processes  to  develop  this  portion  of  their  GWOT  budget  requests. 
Neither  DOD  Comptroller  guidance  nor  DOD  financial  management 
regulations  prescribe  the  use  of  any  particular  method  of  developing 
estimates  for  these  categories.  A  2006  memo  from  the  Deputy  Secretary  of 
Defense  expanded  allowable  costs  for  GWOT  in  several  categories, 
especially  reset-related  procurement  and  equipment  maintenance.  “  These 
types  of  costs  accounted  for  about  70  percent  of  the  total  GWOT  budget 
request  for  fiscal  year  2008,  which  is  a  significant  increase  over  previous 
years.  Due  to  this  increase  in  costs  outside  the  tool,  the  COST-related 
portion  of  GWOT  budget  requests  fell  from  about  80  percent  to  about 


^’Deputy  Secretary  of  Defense  Memorandum  Ground  Rules  and  Process  for  Fiscal  Year 
2007  Spring  Supplemental  (Oct.  25,  2006). 
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30  percent  of  the  total,  although  the  amount  of  the  COST-generated 
estimate  remains  stable  at  between  $40  billion  and  $55  billion  per  year. 


The  Army  Relies  More  on 
COST  to  Develop 
Personnel  and  Operations 
Estimates,  but  the  Other 
Services  Rely  on  Their 
Own  Estimates 


The  Army  uses  COST  as  intended  by  relying  on  the  tool  to  generate  an 
estimate  for  many  personnel  and  operations  costs  for  GWOT  budget 
requests.  During  the  Army’s  development  of  an  estimate  using  COST, 
minor  adjustments  to  COST’S  standard  settings  are  made  to  better  match 
realities  on  the  ground.  For  example,  an  official  developing  an  estimate  in 
COST  might  reduce  the  costs  for  the  transportation  of  equipment  if  a  unit 
will  be  using  equipment  already  in  theater  instead  of  taking  equipment 
with  them.  DOD  officials  and  IDA  representatives  stated  that  COST  is 
better  suited  for  Army  ground  forces;  therefore,  the  Army  relies  on  the 
final  estimate  developed  by  COST  and  submits  this  information  to  the 
DOD  Comptroller  as  part  of  its  GWOT  budget  request.  About  40  to  45 
percent  of  the  Army’s  final  GWOT  budget  requests  are  typically  for  the 
operation  and  maintenance  category  of  appropriations,  and  an  Army 
budget  office  official  stated  that  the  majority  of  this  portion  is  estimated 
by  COST.  Army  officials  further  stated  that  COST  is  an  effective  tool  for 
cost  estimation  because  it  is  frequently  updated  with  cost  data  the  Army 
submits  to  IDA.  An  Army  model  and  database  that  contain  cost 
information  for  personnel  and  equipment  are  the  sources  for  much  of  the 
Army-related  data  used  by  COST  and  also  are  primary  sources  for 
developing  the  Army’s  base  budget  requests. 


The  Air  Force,  Marine  Corps,  and  Navy  all  fulfill  the  requirement  to 
develop  an  estimate  for  their  respective  service’s  personnel  and  operations 
funding  requirements  for  GWOT  using  COST;  however,  these  services 
significantly  alter  the  COST  results  to  match  estimates  they  have 
developed  outside  COST,  using  historical  obligation  data  and  other 
information.  Officials  from  the  Air  Force,  Marine  Corps,  and  Navy  budget 
offices  reported  various  concerns  regarding  the  functionality  and  accuracy 
of  COST  as  reasons  for  relying  more  on  historical  obligation  data  and 
other  information.  Several  budget  officials  from  each  of  these  services 
reported  that  COST  routinely  overestimated  some  costs.  As  a  result,  most 
of  the  changes  they  make  to  COST  results,  based  on  historical  obligation 
data,  are  decreases  in  the  amount  estimated  by  COST.  For  example,  a 
service  budget  official  stated  that  for  one  fiscal  year’s  GWOT  budget 
request,  the  total  COST-developed  estimate  was  $100  million  more  than 
the  estimate  developed  by  the  service  for  the  same  types  of  costs  using 
historical  obligation  data.  Specifically,  COST  overestimated  transportation 
costs  by  about  $275  million,  while  underestimating  certain  personnel 
support  costs  by  about  $200  million,  among  other  discrepancies.  Navy 
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officials  stated  that  COST  often  overestimated  some  types  of 
transportation  costs  and  the  results  had  to  be  manually  adjusted  to  match 
historical  obligation  data  or  other  information.  Similarly,  Marine  Corps 
officials  reported  that  most  adjustments  they  make  to  the  COST  output 
result  in  a  decrease  of  the  estimate.  An  Air  Force  budget  official  stated 
that  COST  overestimated  some  transportation  costs  by  about  $1  billion  in 
a  prior  year’s  estimate,  while  other  costs  were  not  captured  and  therefore 
underestimated.  While  these  discrepancies  were  adjusted  prior  to 
submission  as  a  GWOT  budget  request,  this  official  stated  that  an  IDA 
representative  had  since  suggested  strategies  to  develop  a  more  accurate 
transportation  estimate  in  future  uses  of  the  tool.  Furthermore,  a  Navy 
official  stated  that  COST  is  unable  to  project  certain  costs  associated  with 
civilians,  and  hence  might  underestimate  the  total  costs  due  to  this 
exclusion.  For  example,  the  official  stated  that  COST  does  not 
automatically  estimate  costs  for  civilian  support  positions  associated  with 
an  operational  unit,  such  as  a  ship  or  ground  unit.  Aside  from  accuracy 
concerns,  officials  from  the  Air  Force,  Marine  Corps,  and  Navy  reported 
that  COST  and  the  cost  breakdown  structure  that  forms  the  basis  of 
cost’s  organization  and  resulting  estimations  better  represent  Army 
ground  forces  than  the  unique  characteristics  of  the  other  forces.  Navy 
officials  stated  that  COST  automatically  estimates  food,  ice,  and  water  for 
all  units  because  deployed  ground  forces  require  these  items.  However, 
the  Navy  funds  these  items  for  sailors  on  deployed  ships  through  base 
budget  funding  because  these  costs  are  incurred  regardless  of  a  ship’s 
location.  The  tool  has  not  been  refined  to  accurately  estimate  these  costs; 
therefore.  Navy  officials  must  manually  remove  these  types  of  costs  from 
an  estimate  for  GWOT  funding.  Several  service  officials  stated  that, 
because  of  the  limitations  to  COST,  the  required  process  of  using  COST  to 
develop  a  cost  estimate  was  duplicative  of  their  preferred  method  of  using 
historical  obligation  data  and  other  information  better  suited  to  their 
specific  service  to  develop  a  GWOT  budget  request. 

Service  officials  stated  that  COST  is  a  useful  tool  for  estimating  costs  for 
small-scale  and  short-duration  operations  or  for  situations  for  which 
information  is  unknown  or  new,  such  as  the  recent  troop  surge  or  for 
other  general  rough-order-of-magnitude  estimates  produced  early  in 
operation  planning  while  options  are  being  weighed  by  decision  makers. 
However,  officials  stated  the  tool  does  not  perform  as  well  for  estimating 
costs  for  the  lengthy  deployment  and  sustainment  phases  associated  with 
a  large  campaign  such  as  GWOT.  Furthermore,  COST  is  not  able  to 
estimate  all  costs  associated  with  GWOT. 


Page  12 


GAO-08-982  Defense  Budget 


Military  Services  Use 
Other  Methods  to 
Determine  Cost  Estimates 
for  Procurement,  Most 
Equipment  Maintenance, 
and  Some  Contract  Costs 


Because  COST  does  not  have  the  capability  to  estimate  costs  such  as 
procurement,  reset-related  equipment  maintenance,  and  contracted  needs 
and  services,  service  budget  officials  report  using  historical  obligation 
data,  other  models  and  formulas,  and  other  information,  such  as 
deployment  information,  to  estimate  these  costs.  Reset-related 
procurement  estimates  for  GWOT  are  devised  in  multiple  ways.  For 
example,  to  estimate  costs  to  procure  new  equipment  to  replace  lost  or 
damaged  equipment,  officials  stated  that  incident  reports  are  tracked  to 
provide  information  on  how  many  pieces  of  equipment  are  needed  to 
replace  battle  losses.  For  procurement  to  replace  equipment  that  has 
reached  the  end  of  its  useful  life  because  of  GWOT’s  higher  operating 
tempo,  formulas,  based  on  historical  data,  provide  information  on  the 
normal  extent  of  wear  and  tear  for  certain  types  of  equipment,  and  wear 
above  the  normal  extent  is  attributed  to  GWOT.^^  Most  types  of  equipment 
maintenance  are  not  estimated  by  COST,  such  as  intermediate-  and  depot- 
level  maintenance;  therefore  the  services  again  rely  on  historical 
obligation  data  to  develop  estimates  for  the  GWOT-related  costs.  For 
example.  Army  logistics  officials  track  the  units  that  are  scheduled  for 
redeployment  and  develop  estimates  for  the  cost  of  resetting  a  particular 
unit’s  equipment  based  on  the  type  of  brigade  the  unit  is  part  of,  such  as  a 
heavy  brigade  combat  team  or  Stryker  brigade  combat  team,  and  the 
average  cost  of  resetting  that  type  of  brigade  unit  developed  from 
historical  obligation  data,  adjusted  for  inflation.  For  other  items  outside 
the  scope  of  COST — such  as  the  Logistics  Civil  Augmentation  Program 
costs  for  the  Army,  intelligence  needs,  and  contracts  for  other  needs  such 
as  linguists — functional  experts  within  each  service  provide  the  service 
budget  offices  with  information  to  develop  estimates.  This  information  is 
based  on  contract  task  orders  or  needs  assessments  developed  by  in¬ 
theater  commanders.  For  example,  in-theater  commanders  submit 
requests  for  additional  linguists  to  Army  intelligence  officials,  and  the 
estimated  cost  for  these  linguists  is  developed  based  on  historical  costs  for 
the  same  type  of  linguist.  Army  budget  officials  stated  that  contracting 


^^During  the  course  of  this  review,  we  did  not  validate  or  assess  the  appropriateness  or 
effectiveness  of  the  processes  the  services  use  to  estimate  reset-related  funding  needs.  We 
have  reported  that  the  Army  and  Marine  Corps  cannot  be  assured  that  their  reset  strategies 
will  sustain  equipment  availability  for  units  deployed  in  or  preparing  to  deploy  to  Iraq  and 
Afghanistan  while  meeting  ongoing  operational  requirements  and  also  that  the  Army  and 
Marine  Corps  did  not  report  detailed  reset  expenditures  within  the  procurement  accounts 
in  a  way  that  confirms  that  funds  appropriated  for  reset  were  obligated  and  expended  for 
reset.  See  GAO,  Defense  Logistics:  Army  and  Marine  Corps  Cannot  Be  Assured  That 
Equipment  Reset  Strategies  Will  Sustain  Equipment  Availability  While  Meeting  Ongoing 
Operational  Requirements,  GAO-07-814  (Washington,  D.C.:  Sept.  19,  2007). 
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costs,  including  the  Logistics  Civil  Augmentation  Program,  linguist,  and 
security  services,  are  one  of  the  most  expensive  cost  categories  that  falls 
outside  of  COST.  Officials  from  all  services  stated  that,  after  many  years  of 
ongoing  operations  in  support  of  GWOT,  they  believe  few  requirements 
are  truly  unknown  or  based  on  emerging  needs,  so  therefore  they  are 
comfortable  relying  on  historical  obligation  data  and  other  information  to 
develop  estimates  for  these  types  of  costs.  For  an  example  of  how  DOD 
develops  a  GWOT  budget  request  and  the  use  of  COST  and  other  methods 
of  developing  an  estimate,  see  appendix  111. 


COST  Is  Frequently 
Updated  and  Refined, 
but  Could  Benefit 
from  a  Review  of 
Adherence  to  Best 
Practices 


DOD  has  taken  steps  to  improve  the  performance  and  reliability  of  COST; 
however,  COST  could  benefit  from  a  review  of  the  tool’s  adherence  to  best 
practices  for  high-quahty  cost  estimation  as  outlined  in  our  Cost 
Assessment  Guide.  Revisions  have  been  made  to  the  tool  to  improve  its 
performance,  and  frequent  updates  are  made  to  the  data  used  by  the  tool. 
However,  a  review  of  COST  according  to  the  best  practices  for  cost 
estimation  could  provide  decision  makers  with  information  on  the  extent 
to  which  the  tool  generates  reliable  estimates  and  identify  opportimities 
for  improvement.  Our  guide  defines  high-quality  cost  estimates  as  well 
documented,  comprehensive,  accurate,  and  credible.  While  we  did  not 
undertake  a  full  assessment  of  COST  against  best  practices,  during  the 
course  of  our  review  we  identified  features  of  COST’S  estimation  process 
that  meet  best  practices  and  other  features  that  would  benefit  from  further 
review.  For  example,  COST  adheres  to  several  best  practices  for  a 
comprehensive  and  accurate  cost  estimate,  such  as  frequent  updates  to 
the  structure  of  COST  and  the  data  that  COST  uses  to  generate  estimates. 
While  COST  appears  to  largely  encompass  the  types  of  costs  that  are 
incurred  to  deploy  and  sustain  Army  ground  forces  and  their  related 
equipment,  COST  may  not  comprehensively  and  accurately  estimate  costs 
for  the  other  services.  COST  also  relies  on  GWOT  obligation  data  from 
DOD’s  Supplemental  and  Cost  of  War  Execution  Reports  that  we  have 
identified  as  being  of  questionable  reliability,  and  DOD  is  taking  steps  to 
improve. “  These  might  be  areas  for  which  a  thorough  and  full  review  of 


^^See  GAO,  Global  War  on  Terrorism:  DOD  Needs  to  Improve  the  Reliability  of  Cost  Data 
and  Provide  Additional  Guidance  to  Control  Costs,  GAO-06-882  (Washington,  D.C.:  Sept. 
21,  2005),  Global  War  on  Terrorism:  Fiscal  Year  2006  Obligation  Rates  Are  Within 
Funding  Levels  and  Significant  Multiyear  Procurement  Funds  Will  Likely  Remain 
Available  for  Use  in  Fiscal  Year  2007,  GAO-07-76  (Washington,  D.C.:  Nov.  13,  2006),  and 
GAO-08-68. 
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the  tool  could  improve  the  resulting  estimates  that  are  used  to  develop 
GWOT  budget  requests. 

COST  Revised  and 
Frequently  Updated  in  an 
Effort  to  Improve 
Effectiveness 

IDA  has  made  changes  to  the  structure  of  COST,  either  at  the  request  of 
the  DOD  Comptroller  or  the  services,  or  on  its  own  initiative,  and  has 
refined  the  tool  many  times  over  the  past  several  years  to  improve 
functionality  or  performance.  Recent  refinements  to  COST  included 
changes  that  provide  the  ability  to  alter  the  percentage  of  officer  and 
enlisted  personnel  within  a  unit  and  the  types  of  diagnostic  and  summary 
reports  that  the  tool  can  create.  A  3-year  development  effort  culminated  in 
June  2007  with  the  release  of  a  new  version  of  COST  supported  by  new 
software  and  hardware  that  increased  functionality  and  performance.  This 
new  version  allows  the  user  to  simultaneously  develop  estimates  for 
multiple  operations  within  the  same  contingency.  The  services  and  others 
frequently  submit  new  information  for  tool  updates.  For  example, 
components  are  asked  to  submit  updated  cost  factor  data  prior  to  the 
development  of  any  request  for  supplemental  emergency  funding. 
Additionally,  the  DOD  Comptroller  has  asked  IDA  to  review  the  inputs, 
assumptions,  and  processes  the  services  used  to  generate  COST  estimates 
for  GWOT  budget  requests  since  fiscal  year  2005.  The  reviews  revealed 
issues  that  were  consistent  across  the  services  or  significant  enough  to 
warrant  attention  in  future  use  of  COST.  For  example,  a  review  found  that 
COST  users  estimated  an  excessive  use  of  airlift  for  the  movement  of 
cargo  with  no  scheduled  cargo  for  the  return  flight.  Additionally,  a  review 
identified  confusion  regarding  the  use  of  different  operational  tempo 
factors  and  pay  offsets.  IDA  consolidated  these  and  other  issues  identified 
in  the  review  of  fiscal  years  2007  and  2008  into  a  lessons  learned  briefing 
and  checkhst  to  assist  the  services  as  they  use  COST  in  the  development 
of  future  GWOT  budget  requests. 

COST  Could  be  Assessed 
for  Adherence  to  Cost 
Estimation  Best  Practices 

While  DOD  has  taken  steps  to  revise  and  update  COST  to  improve 
effectiveness,  COST  has  not  been  assessed  according  to  best  practices  for 
cost  estimation  that  define  reliable,  high-quality  cost  estimates.  A  review 
of  regulations,  guidance,  and  best  practices  for  cost  estimation  and  best 
practices  established  by  professional  cost  analysts,  and  compiled  in  our 
Cost  Assessment  Guide,  identified  four  characteristics  of  high-quahty, 
reliable  cost  estimates.^  As  shown  in  table  1,  cost  estimates  should  be  well 

^“^See  GAO-07-1134SP  and  OMB,  Guidelines  and  Discount  Rates  for  Benefit-Cost  Analysis 
of  Federal  Programs,  Circular  No.  A-94  Revised  (Washington,  D.C.:  Oct.  29,  1992). 
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documented,  comprehensive,  accurate,  and  credible.  DOD  Comptroller 
officials  stated  they  are  confident  in  the  tool’s  ability  to  provide  reasonable 
estimates  because  COST  is  frequently  updated.  However,  neither  DOD,  nor 
any  other  entity,  has  assessed  COST  and  its  resulting  cost  estimates 
against  these  best  practices.  While  we  did  not  perform  a  full  assessment  of 
COST  against  best  practices,  during  the  course  of  our  work  we  identified 
some  features  of  COST  that  meet  best  practices  for  cost  estimation  and 
other  areas  that  could  benefit  from  further  review. 

•  Well  Documented:  A  well  documented  cost  estimate  is  based  on  data 
that  have  been  gathered  from  actual  historical  costs  and  technical 
experts,  analyzed  for  cost  drivers,  and  collected  from  primary  sources. 
These  best  practices  appear  to  be  met  by  COST.  Additionally,  any 
adjustments  made  to  COST’S  standard  settings  are  flagged  and  must  be 
accompanied  by  an  explanatory  note  that  details  why  information  was 
changed,  and  these  situations  are  reviewed  by  DOD  Comptroller 
officials.  Furthermore,  best  practices  also  require  that  previous  cost 
factors  and  data  are  stored  after  updates  so  that  an  estimation  process 
is  repeatable  and  can  be  later  verified.  The  newest  version  of  COST 
does  have  this  capability  and  IDA  maintains  records  of  cost  factors  and 
other  data.  However,  best  practices  also  require  that  data  used  in  a 
model  should  be  traced  back  to  the  source  documentation  and  any 
normalization  steps  should  be  documented.  The  services  are 
responsible  for  ensuring  data  are  reliable  and  neither  IDA  nor  DOD, 
including  the  services,  traces  all  data  back  to  the  source  documents. 
Additionally,  IDA  officials  stated  that  unusually  high  or  low  data  might 
be  removed  and  no  record  of  these  actions  would  be  kept,  and  best 
practices  require  that  these  sorts  of  steps  should  be  documented. 
Further  review  could  reveal  if  these  or  other  areas  might  need  more 
work  to  ensure  the  estimate  is  well  documented. 

•  Comprehensive:  Estimates  for  personnel  and  operations  costs 
developed  by  COST  appear  to  meet  several,  but  not  all,  of  the  criteria 
for  comprehensive  cost  estimates.  For  example,  the  cost  breakdown 
structure,  which  defines  the  cost  elements  within  COST  and  forms  the 
foundation  of  formulas  within  the  tool,  has  more  than  three  levels  of 
detail,  the  structure  is  updated  as  changes  occur,  and  each  element  is 
defined  in  a  cost  breakdown  structure  dictionary  included  in  the 
financial  management  regulation  for  contingency  operations.  These 
steps  are  all  considered  best  practices  for  a  comprehensive  cost 
estimate.  However,  our  analysis  of  the  cost  breakdown  structure  in  the 
financial  management  regulation  revealed  that  there  are  some  errors 
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and  ambiguities  in  the  structure’s  definitions  that  might  allow  for 
double  counting  of  costs. Furthermore,  while  COST  appears  to  largely 
encompass  the  types  of  costs  that  are  incurred  to  deploy  and  sustain 
Army  ground  forces  and  their  related  equipment,  it  may  not 
comprehensively  estimate  some  of  the  types  of  costs  incurred  by  the 
other  services.  For  example,  COST  does  not  develop  an  estimate  for 
the  costs  of  certain  Navy  ground  support  units,  such  as  intelligence, 
that  are  not  associated  with  a  naval  fleet.  Moreover,  COST  is  not  used 
by  the  Air  Force  to  develop  estimates  for  the  cost  of  transporting 
people  in  non-combat  situations,  such  as  the  transport  of  military  or 
civilian  personnel  from  the  International  Zone  to  a  forward  operating 
base,  for  example.  An  Air  Force  official  stated  that  this  is  because  the 
tool  automatically  assumes  that  any  flying  hour-related  expenses  are 
operational  or  combat-related  and  these  costs  should  instead  be 
attributed  to  transportation-related  expenses.  Further  review  could 
identify  cost  data  or  formulas  within  the  tool  that  could  be  refined  to 
better  suit  the  other  services  or  might  reveal  some  types  of  costs  for 
which  COST  should  not  be  used  to  develop  an  estimate.  Additionally, 
the  tool  does  not  comprehensively  estimate  all  GWOT  costs,  such  as 
procurement  and  many  types  of  equipment  maintenance.  The  GWOT 
cost  estimate  presented  in  appendix  III  illustrates  the  revisions  made  to 
COST  results  and  the  types  of  costs  that  were  estimated  outside  of 
COST  for  a  particular  estimate  that  was  to  be  comprised  primarily  of 
military  personnel  and  operations  costs. 

•  Accurate:  Similarly,  the  tool’s  estimates  appear  to  meet  some,  but  not 
all,  of  the  best  practices  for  accuracy.  For  example,  an  accurate 
estimate  should  be  based  on  cost  factors  that  reflect  updates  and 
changes.  IDA  does  ask  the  components  to  submit  updated  cost  factors, 
which  are  sometimes  based  on  historical  obligation  data,  prior  to  every 
run  of  the  tool  for  a  fiscal  year’s  GWOT  request,  but  the  components 
are  not  required  to  update  the  factors.  IDA  reviews  the  cost  factor 
submission  to  ensure  general  consistency  across  years,  but  the  services 
and  other  components  that  submit  data  are  ultimately  responsible  for 
the  data  and  are  not  required  to  validate  the  data  prior  to  their 
inclusion  into  the  tool.  According  to  IDA  representatives,  IDA  does  not 
validate  the  data  sources  or  obtain  assurances  that  the  data  submitted 
are  reliable,  because  this  requirement  is  not  included  in  its  contract 


^^For  example,  two  categories  are  defined  as  including  costs  related  to  the  logistics  civil 
augmentation  program  (LOGCAP):  both  the  category  for  facilities  and  base  support  and  a 
separate  category  for  LOGCAP.  DOD  stated  in  its  technical  comments  regarding  a  draft  of 
this  report  that  this  issue  has  been  reviewed  and  is  currently  under  revision. 
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with  DOD.  The  services  do  not  validate  all  data  submitted  to  IDA  for 
updates,  nor  do  they  submit  updates  for  all  factors  that  need  updating. 
Additionally,  some  cost  factors  are  developed  from  the  Cost  of  War 
reports  and  other  data  are  compared  against  these  reports  as  a 
validation  check.  These  practices  might  benefit  from  review  since  our 
previous  work  has  raised  concerns  about  the  reliability  of  reported 
GWOT  obligation  data.  For  example,  we  have  reported  that  there  is  a 
lack  of  transparency  over  certain  obligations  in  the  Cost  of  War  reports 
and  we  have  identified  inaccuracies  in  these  reports.  Consequently,  we 
were  unable  to  ensure  that  DOD’s  reported  obligations  for  GWOT  are 
complete,  reliable,  and  accurate,  and  believe  that  they  should  therefore 
be  considered  approximations.  However,  we  acknowledge  that  DOD 
has  taken  steps  to  address  our  recommendations  and  the  department 
has  several  initiatives  underway  to  further  improve  the  reliability  of 
GWOT  obligation  data.  Additionally,  according  to  service  officials, 
many  adjustments  to  COST  results  for  transportation  and  other  types 
of  costs  are  made  to  decrease  estimates.  This  raises  concerns  regarding 
the  accuracy  or  applicability  of  some  data  in  COST. 

•  Credible:  Many  of  the  best  practices  associated  with  a  credible  cost 
estimate  require  sensitivity  analysis,  risk  and  uncertainty  analysis,  and 
a  comparison  against  an  independent  cost  estimate  to  be  performed.  “ 
Sensitivity  analysis  has  been  performed  to  understand  cost  drivers  and 
this  type  of  analysis  can  be  performed  during  estimate  development  as 
characteristics  of  the  operation  change.  However,  IDA  representatives 
stated  that  risk  analysis  is  unnecessary  since  the  largest  sources  of  risk 
stem  from  the  changing  and  unpredictable  nature  of  warfare  and  policy 
changes  that  also  cannot  be  predicted.  Additionally,  uncertainly 
analysis  can  assess  the  impact  that  the  variability  of  certain  unknown 
factors  will  have  on  resulting  estimates.  For  example,  uncertainty 
analysis  of  possible  fuel  price  changes  could  result  in  various  cost 
scenarios  that  might  occur  depending  on  future  fuel  prices.  Finally, 
cost  results  should  be  compared  to  an  independent  cost  estimate  which 
is  another  best  practice  for  a  credible  cost  estimation  process.  A  Joint 
Staff  or  other  official  will  often  develop  a  COST  estimate  to  compare 
against  an  estimate  developed  by  a  service  official  to  ensure  the 
appropriate  assumptions  were  used.  A  thorough  assessment  against 


^^Sensitivity  analysis  can  identify  key  elements  that  drive  cost  and  provide  information  to 
decision  makers  regarding  the  potential  for  cost  increases  and  reasons  for  increases.  Risk 
and  uncertainty  analysis  assess  the  variability  in  the  cost  estimate.  Specifically,  uncertainty 
analysis  provides  perspective  on  the  potential  variability  of  the  estimate  should  facts, 
circumstances,  and  assumptions  change. 
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best  practices  would  reveal  if  these  or  other  areas  might  need  more 
work  to  ensure  the  estimate  is  sufficiently  credible. 


Conclusions 


Estimating  needs  and  costs  that  will  occur  in  the  future  is  not  an  exact 
science,  and  the  use  of  cost  estimation  tools  or  historical  obligation  data, 
along  with  other  factors,  can  be  reasonable  means  of  projecting  future 
costs.  However,  due  to  the  significant  and  ever-increasing  size  of  GWOT 
budget  requests,  every  attempt  should  be  made  to  ensure  that  a  cost 
estimation  process  is  sound.  COST  has  been  used  to  generate  hundreds  of 
billions  of  dollars  in  budget  requests  for  GWOT  over  the  past  several  years 
and  the  services  are  required  to  use  it.  While  the  tool  and  underlying  data 
have  been  refined  and  updated,  GOST’s  overall  effectiveness  for  estimating 
GWOT  costs  has  not  been  assessed.  Best  practices  criteria  to  measure 
GOST’s  extent  of  sufficient  documentation,  comprehensiveness,  accuracy, 
and  credibility  could  provide  additional  information  about  the  tool’s 
effectiveness  at  generating  high-quality  cost  estimates  and  steps  of  the 
process  that  could  be  further  improved.  Without  a  thorough  review  of 
COST  according  to  these  best  practices,  decision  makers  within  both  DOD 
and  Congress  cannot  be  assured  that  estimates  generated  by  the  tool  are 
developed  using  valid  data  and  sound  processes.  Officials  from  across 
DOD  and  the  services  report  that  COST  performs  well  for  predicting 
budget  needs  for  ground  forces  and  for  small-scale  operations  of  short 
duration,  or  for  situations  in  which  detailed  information  is  unknown. 
However,  in  light  of  service  officials’  concerns  that  COST  does  not 
perform  as  well  for  their  needs  and  might  generate  estimates  that  are  too 
high  in  certain  areas,  it  is  important  that  DOD  review  the  applicability  of 
COST  for  all  of  the  services  or  investigate  ways  in  which  to  make  the  tool 
better  suit  the  needs  of  all  the  services. 


Recommendations 
Executive  Action 


J'qj,  To  ensure  that  DOD  budget  requests  for  GWOT  are  based  on  a  sound  cost 
estimation  process,  we  recommend  that  the  Secretary  of  Defense  direct 
the  Under  Secretary  of  Defense  (Comptroller)  to 


•  Arrange  for  an  independent  review  of  COST  against  best  practices  for 
cost  estimation. 
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Based  on  that  review,  and  taking  into  consideration  how  each  service  uses 
COST,  we  recommend  that  the  Secretary  of  Defense  direct  the  Under 
Secretary  of  Defense  (Comptroller)  to 

•  Consider  options  for  refining  COST,  determine  the  appropriate  items  or 
types  of  costs  for  which  COST  should  be  applied,  and  identify  methods  to 
be  used  when  COST  is  not  appropriate. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  agreed  with  both  of 
our  recommendations  for  executive  action.  These  written  comments 
additionally  provided  examples  of  steps  DOD  has  taken  or  plans  to  take 
that  it  considers  actions  that  address  aspects  of  our  recommendations. 
Also,  DOD  provided  us  with  technical  comments  which  we  incorporated  in 
the  report  where  appropriate.  DOD’s  comments  are  reprinted  in 
appendix  IV. 

DOD  agreed  with  our  recommendation  that  the  DOD  Comptroller  arrange 
for  an  independent  review  of  the  Contingency  Operations  Support  Tool 
(COST)  against  best  practices  for  cost  estimation.  In  its  comments  on  this 
recommendation,  DOD  agreed  with  the  concept  of  an  independent  review 
of  the  tool  against  best  practices,  and  further  noted  that  the  Air  Force 
Studies  and  Analyses  Agency  (AFSAA)  conducted  a  review  of  COST’S  use 
in  developing  cost  estimates  for  the  air  war  over  Serbia.  This  review 
compared  the  tool’s  output  against  the  actual  reported  costs  compiled  by 
the  Defense  Finance  and  Accounting  Service.  DOD  stated  that  the 
recommendations  for  improvement  identified  in  the  review  were 
incorporated  into  the  tool.  While  the  review  and  the  incorporation  of  its 
recommendations  into  the  tool  are  positive  steps,  the  review  did  not 
assess  COST’S  use  in  estimating  the  broad  scope  of  costs  that  are 
associated  with  a  large-scale  campaign  such  as  GWOT.  For  example,  the 
scope  of  the  AFSAA  review  of  COST  was  limited  to  the  3-month  air  war 
over  Serbia  and  primarily  reviewed  costs  of  the  Air  Force.  Furthermore, 
the  AFSAA  review  was  not  a  thorough  review  against  best  practices  for 
cost  estimation,  which  requires  cost  estimates  to  be  well  documented, 
comprehensive,  accurate,  and  credible.  Therefore,  we  continue  to 
recommend  that  an  independent  and  thorough  review  of  the  tool  against 
best  practices  for  cost  estimation  be  pursued.  This  type  of  review  would 
include  an  assessment  of  the  risk  and  uncertainty  associated  with  the 
inputs  to  the  tool  and  the  accuracy  of  the  underlying  equations  and  data 
the  tool  relies  on  to  estimate  costs.  DOD  additionally  stated  that 
(1)  cost’s  factors,  processes,  and  algorithms  are  updated  as  needed  and 
the  tool  is  updated  to  reflect  changes  to  congressionally  determined 
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factors,  such  as  military  pay  rates,  and  (2)  COST  relies  on  many  of  the 
same  service-specific  cost  factors  that  are  used  during  the  development  of 
baseline  budgets.  While  we  acknowledge  in  this  report  that  factors  and 
other  aspects  of  the  model  are  regularly  updated,  these  actions  are  a  check 
for  accuracy  by  the  users  of  the  data  and  our  recommendation  specifically 
calls  for  an  independent  review  of  COST.  Finally,  DOD  stated  that,  due  to 
our  recommendation,  the  DOD  Comptroller  will  issue  guidance  that  will 
be  incorporated  into  DOD’s  financial  management  regulation  that  includes 
a  process  for  updating  COST  to  ensure  it  reflects  the  most  current 
budgetary  assumptions  and  a  process  for  evaluating  the  functionality  of 
the  model  to  determine  if  adjustments  are  needed.  As  this  revision  of  the 
financial  management  regulation  has  not  been  finalized,  we  did  not  assess 
this  planned  action.  However,  this  positive  step,  once  completed,  should 
be  taken  into  account  as  part  of  an  independent  and  thorough  review  of 
COST  against  best  practices  for  cost  estimation. 

DOD  agreed  with  our  recommendation  that,  based  on  an  independent 
review,  it  should  consider  options  for  refining  COST,  determine  the 
appropriate  items  or  types  of  costs  for  which  COST  should  be  applied,  and 
identify  methods  to  be  used  when  COST  is  not  appropriate.  In  its 
comments,  DOD  stated  that  it  uses  every  opportunity  to  refme  and 
improve  the  COST  model,  such  as  the  multi-level  review  of  COST  results 
and  assumptions  during  the  development  of  GWOT  budget  requests. 
Additionally,  changes  to  COST  are  made  based  on  training  sessions  and 
feedback  from  COST  users.  Finally,  DOD  stated  that  an  extensive  review 
process  is  in  place  for  each  non-COST  line  item  of  GWOT  budget  requests. 
While  we  did  not  assess  the  process  DOD  uses  to  review  non-COST  line 
items,  our  report  acknowledges  that  DOD  has  refined  and  updated  COST 
many  times  as  information  has  changed  and  the  needs  of  the  department 
have  evolved.  However,  we  reiterate  our  view  that  the  tool  should  be 
subject  to  an  independent  review,  and  COST  should  be  further  refined 
based  on  the  findings  of  that  review,  as  needed. 


We  are  sending  copies  of  this  report  to  other  interested  congressional 
committees;  the  Secretary  of  Defense;  the  Under  Secretary  of  Defense 
(Comptroller),  and  the  Director,  Office  of  Management  and  Budget.  Copies 
of  this  report  will  also  be  made  available  to  others  upon  request.  In 
addition,  this  report  will  be  available  at  no  charge  on  the  GAO  Web  site  at 
http  ://www.gao.  gov. 
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If  you  have  any  questions  regarding  this  report,  please  contact  Sharon 
Pickup  at  (202)  512-9619  or  pickups@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the 
last  page  of  this  report.  GAO  staff  who  made  major  contributions  to  this 
report  are  listed  in  appendix  V. 


Sharon  L.  Pickup 

Director,  Defense  Capabilities  and  Management 
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List  of  Congressional  Committees 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Kent  Conrad 
Chairman 

The  Honorable  Judd  Gregg 
Ranking  Member 
Committee  on  the  Budget 
United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ike  Skelton 
Chairman 

The  Honorable  Duncan  L.  Hunter 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  John  M.  Spratt,  Jr. 
Chairman 

The  Honorable  Paul  Ryan 
Ranking  Member 
Committee  on  the  Budget 
House  of  Representatives 
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The  Honorable  John  P.  Murtha 
Chairman 

The  Honorable  C.W.  Bill  Young 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Scope  and  Methodology 


To  assess  how  the  Department  of  Defense  (DOD)  uses  the  Contingency 
Operations  Support  Tool  (COST)  and  other  processes  to  develop  Global 
War  on  Terror  (GWOT)  budget  requests,  we  reviewed  and  analyzed 
relevant  documents  and  interviewed  key  DOD  and  service  officials  and 
representatives  from  the  Institute  for  Defense  Analyses  (IDA).  Documents 
that  we  used  for  our  review  included,  but  were  not  limited  to,  (1)  relevant 
DOD  directives,  instructions,  and  memoranda  related  to  budgeting 
processes;  (2)  DOD  financial  management  regulations  that  provide  policy 
and  procedures  for  contingency  operations;  (3)  DOD  guidance  for  the 
preparation  and  submission  of  requests  for  incremental  funding  for 
GWOT;  and  (4)  service  budget  office  guidance  for  the  preparation  and 
submission  of  GWOT  budget  requests.  We  obtained  testimonial  evidence 
from  officials  representing  the  Office  of  the  Under  Secretary  of  Defense 
(Comptroller)  and  the  Joint  Staff  regarding  the  processes  used  to  develop 
GWOT  budget  requests  and  the  role  of  COST  in  that  process.  Specifically, 
we  obtained  their  perspectives  on  COST’S  effectiveness  and  accuracy,  as 
well  as  the  processes  employed  to  develop  estimates  for  costs  that  are 
outside  the  scope  of  the  tool’s  estimating  capabilities.  We  similarly 
interviewed  key  service  officials  in  the  financial  management  or  budget 
office  responsible  for  developing  GWOT  cost  estimates  for  contingency 
operations  in  the  Air  Force,  Army,  Marine  Corps,  and  Navy  to  understand 
their  experiences  using  COST  during  the  development  of  prior  years’ 
GWOT  budget  requests,  including  strengths  and  weaknesses  of  the  tool. 
We  additionally  interviewed  service  officials  that  were  identified  as 
functional  experts  for  the  types  of  costs  that  must  be  estimated  outside  of 
the  tool,  such  as  procurement,  reset-level  equipment  maintenance,  and 
intelligence  needs.  We  discussed  what  processes  they  use  to  develop 
estimates  for  these  types  of  costs.  We  attended  several  briefings  regarding 
COST  presented  by  IDA  and  attended  training  sessions  on  the  tool.  We 
interviewed  IDA  representatives  about  the  tool  and  reviewed  numerous 
briefings  about  COST’S  use  in  cost  estimation,  the  structure  of  the  tool, 
and  the  tool’s  development.  Finally,  in  order  to  understand  how  DOD 
developed  an  actual  GWOT  budget  request  and  the  use  of  COST  and  other 
methods  to  develop  that  estimate,  we  asked  DOD  to  demonstrate  how  an 
estimate  was  developed  for  a  case  study,  which  was  the  $6.3  billion 
estimate  for  military  operations  that  was  included  as  part  of  DOD’s 
October  2007  $42.3  billion  amendment  to  the  Fiscal  Year  2008  GWOT 
supplemental  request  for  emergency  funding.  We  chose  this  example  as 
our  case  study  because  it  was  a  recent  estimate  and  was  assumed  to  be 
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comprised  primarily  of  military  personnel  and  operations  costs,  due  to  the 
description  of  this  estimate  in  the  justification  document  for  the 
amendment  to  the  Fiscal  Year  2008  GWOT  budget  request/  DOD  provided 
the  initial  and  approved  estimate  for  this  request  by  service  and  by 
appropriation  category.  We  discussed  this  estimate  with  DOD  and  service 
officials,  including  the  assumptions  and  processes  that  were  used  to 
develop  this  estimate,  the  reasons  for  changes  between  the  initial  and 
approved  estimates,  and  the  types  of  costs  estimated  by  COST  or  outside 
of  COST.  We  did  not  validate  the  assumptions  used  to  generate  this 
estimate  or  the  data  DOD  presented  in  this  example. 

To  assess  what  actions  DOD  has  taken  to  ensure  COST  adheres  to  best 
practices  for  cost  estimation,  we  reviewed  apphcable  best  practices  and 
compared  DOD’s  efforts  against  those  best  practices  we  found  to  be 
consistently  associated  with  reliable,  high-quality  cost  estimation.^  We 
reviewed  DOD  guidance  regarding  requirements  for  the  verification, 
validation,  and  accreditation  of  tools  and  simulations  used  by  DOD.  We 
reviewed  documents  obtained  from  DOD  and  IDA  and  had  discussions 
with  DOD  officials  and  IDA  representatives  about  COST;  for  example,  we 
reviewed  numerous  briefings  about  COST  and  the  training  manual  that 
documents  the  tool’s  specifications,  development,  and  use  in  detail.  We 
interviewed  DOD  Comptroller  and  IDA  representatives  about 
improvements  and  updates  that  have  been  made  to  COST  and  the  purpose 
of  those  updates.  We  obtained  testimonial  evidence  from  DOD 
Comptroller  officials  and  IDA  representatives  about  the  steps  to  develop 
COST  and  the  processes  that  surround  the  tool’s  use  as  part  of  developing 
a  GWOT  budget  request,  and  identified  steps  that  appeared  to  meet  certain 
criteria  of  established  best  practices  and  those  that  appeared  to  warrant 
further  review.  We  did  not  perform  a  full  review  of  COST  against  all  best 
practices,  but  presented  examples  of  how  the  tool  meets  certain  best 
practices  to  provide  some  context  to  decision  makers  about  what  might  be 
considered  strengths  of  the  tool  and  we  highlighted  some  areas  that  might 
benefit  from  a  full  and  independent  review  of  COST.  These  examples  are 
meant  to  serve  as  illustrative  detail  to  provide  more  information  to 
decision  makers,  but  should  not  be  considered  the  results  of  a  complete, 
thorough,  and  independent  review  of  the  tool. 


^DOD,  Fiscal  Year  2008  Global  War  on  Terror  Amendment  (October  2007). 

^GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estimating  and  Managing  Program 
Costs,  GAO-07-1134SP  (Washington,  D.C.:  July  2007). 
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We  conducted  this  performance  audit  from  July  2007  through 
September  2008  in  accordance  with  generally  accepted  government 
auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 
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Appendix  II:  The  12  Steps  of  High-Quality 
Cost  Estimating  Mapped  to  Best  Practice 
Criteria 


Characteristic 

Weii  documented 

•  The  estimate  is  thoroughly  documented,  including  source  data  and 
significance,  clearly  detailed  calculations  and  results,  and  explanations  for 
choosing  a  particular  method  or  reference. 

•  Data  have  been  traced  back  to  the  source  documentation. 

•  A  technical  baseline  description  is  included. 

•  All  steps  in  developing  the  estimate  are  documented,  so  that  a  cost  analyst 
unfamiliar  with  the  program  can  recreate  it  quickly  with  the  same  result. 

•  All  data  sources  for  how  the  data  were  normalized  are  documented. 

•  The  estimating  methodology  and  rationale  used  to  derive  each  cost 
breakdown  structure  element’s  cost  are  described  in  detail. 

Comprehensive 

•  The  estimate’s  level  of  detail  ensures  that  cost  elements  are  neither  omitted 
nor  double  counted. 

•  All  cost-influencing  ground  rules  and  assumptions  are  detailed. 

•  The  cost  breakdown  structure  is  defined  and  each  element  is  described  in  a 
cost  breakdown  structure  dictionary. 


Related  step 


Define  the  estimate’s  purpose 

Define  the  program 

Identify  ground  rules  and  assumptions 

Obtain  the  data 

Document  the  estimate 

Present  estimate  to  management 


Develop  the  estimating  plan 
Determine  the  estimating  approach 


Accurate 

•  The  estimate  is  unbiased,  not  overly  conservative  or  overly  optimistic,  and 
based  on  an  assessment  of  most  likely  costs. 

•  It  has  few,  if  any,  mathematical  mistakes;  those  it  has  are  minor. 

•  It  has  been  validated  for  errors  like  double  counting  and  omitted  costs. 

•  It  has  been  compared  to  the  independent  cost  estimate  for  differences. 

•  Cross-checks  have  been  made  on  cost  drivers  to  see  if  results  are  similar. 

•  The  estimate  is  timely. 

•  It  is  updated  to  reflect  changes  in  technical  or  program  assumptions  and  new 
phases  or  milestones. 

•  Estimates  are  replaced  with  the  earned  value  management  estimate  at 
completion  and  the  independent  estimate  at  completion  from  the  integrated 
earned  value  management  system. 

Credible 

•  Any  limitations  of  the  analysis  because  of  uncertainty  or  biases  surrounding  Conduct  sensitivity  analysis 

data  or  assumptions  are  discussed.  r-  ^  ^  ^  *  .  ■ 

^  Conduct  risk  and  uncertainty  analysis 

•  Major  assumptions  are  varied  and  other  outcomes  recomputed  to  determine 
how  sensitive  outcomes  are  to  changes  in  the  assumptions. 

•  Risk  and  uncertainty  analysis  is  performed  to  determine  the  level  of  risk 
associated  with  the  estimate. 

•  The  results  are  cross-checked  and  an  independent  cost  estimate  is 
developed  to  determine  if  other  estimating  methods  produce  similar  results. 


Develop  the  point  estimate 

Update  the  estimate  to  reflect  actual  costs  and 
changes 


Source:  GAO. 
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Appendix  III:  An  Example  of  DOD’s  GWOT 
Budget  Estimate  Process  Using  the  Sustainment 
and  Redeployment  of  the  Troop  Surge 


To  better  understand  how  the  Department  of  Defense  (DOD)  develops  a 
Global  War  on  Terrorism  (GWOT)  budget  request  and  the  use  of  the 
Contingency  Operations  Support  Tool  (COST)  and  other  methods  of 
developing  an  estimate,  we  asked  DOD  to  demonstrate  how  the  services 
used  COST  and  other  methods  to  develop  the  $6.3  billion  budget  estimate 
for  military  operations  that  was  included  as  part  of  DOD’s  October  2007 
$42.3  billion  amendment  to  the  fiscal  year  2008  GWOT  supplemental 
request  for  emergency  funding.  DOD  justification  materials  for  the 
amendment  describe  this  $6.3  billion  portion  as  needed  to  support  the 
continued  sustainment  and  redeployment  of  the  five  Army  brigades  and 
two  Marine  Corps  infantry  battalions  that  were  considered  part  of  the 
troop  surge.^  This  portion  of  the  amendment  was  also  requested  to  support 
the  simultaneous  deployment  of  combat  support  forces  augmenting  these 
combat  forces  and  other  costs  related  to  the  presence  of  the  troop  surge. 
Table  3  presents  DOD’s  estimate  for  this  $6.3  billion  portion  of  the 
amendment  meant  to  fund  additional  operations,  broken  into  the  COST- 
generated  portion  and  the  portion  that  was  estimated  by  other  means. 
Typically,  service  budget  officials  develop  cost  estimates  for  GWOT,  but 
officials  stated  that  time  constraints  for  this  specific  estimate  required  that 
Joint  Staff  and  DOD  Comptroller  officials  develop  the  initial  estimate  using 
both  COST  and  other  processes  outside  of  COST  as  necessary.  After 
discussions  with  service  budget  officials,  and  the  September  10,  2007, 
testimony  of  the  Commander  of  the  Multi-National  Force-Iraq  before 
Congress,  the  estimates  were  revised.  This  is  reflected  in  the  approved 
estimate  for  each  service,  again  broken  into  the  COST  portion  and  the 
“non-COST”  portion.  Each  service’s  estimate  is  further  broken  down  into 
estimates  for  Military  Personnel  appropriations  and  Operation  and 
Maintenance  appropriations. 


4n  late  January  2007,  the  President  announced  a  “surge”  strategy  in  Iraq,  providing  for  the 
deployment  of  an  additional  30,000  troops  to  support  stability  operations.  He  also 
announced  the  deployment  of  additional  personnel  to  Afghanistan  to  provide  increased 
security  against  an  anticipated  insurgent  offensive. 
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Sustainment  and  Redeployment  of  the  Troop 
Surge 


Table  3:  Example  of  DOD  Estimation  for  Operations  in  Support  of  GWOT 


Dollars  in  millions 


initial  Estimate  Approved  Estimate 


COST  Estimate 

Non-COST 

Estimate 

Total  Estimate 

COST  Estimate 

Non-COST 

Estimate 

Total  Estimate 

Army  Total 

$4,107 

$775 

$4,882 

$3,918 

$1,837 

$5,755 

Military  Personnel 

731 

0 

731 

705 

(961) 

(256) 

Operation  & 
Maintenance  (O&M) 

3,376 

775 

4,151 

3,213 

2,798 

6,011 

Navy  Total 

$92 

$0 

$92 

$96 

$100 

$196 

Military  Personnel 

10 

0 

10 

10 

30 

40 

O&M 

83 

0 

83 

87 

69 

156 

Marine  Corps  Total 

$25 

$0 

$25 

$146 

$163 

$309 

Military  Personnel 

6 

0 

6 

25 

163 

188 

O&M 

19 

0 

19 

121 

0 

121 

Air  Force  Total 

$103 

$0 

$103 

$19 

$25 

$44 

Military  Personnel 

11 

0 

11 

4 

0 

4 

O&M 

92 

0 

92 

15 

25 

40 

SOCOM 

$10 

$0 

$10 

$7 

$0 

$7 

Total  Defense-Wide 

$4,337 

$775 

$5,112 

$4,186 

$2,125 

$6,311 

Source:  DOD. 

As  shown  in  table  3,  the  Army’s  estimate  of  nearly  $5.8  billion  comprises 
the  majority  of  the  $6.3  billion  total  estimate.  COST-generated  estimates 
for  military  personnel  or  operation  and  maintenance  did  not  change 
significantly  for  the  Army  from  its  initial  estimate  to  the  approved  final 
estimate.  Substantial  changes  were  made,  however,  to  the  portions  of  the 
estimate  that  are  outside  the  scope  of  COST.  Army  budget  officials 
reported  that  the  large  decrease  in  requested  military  personnel  funding 
was  due  to  lower  mobilization  levels  than  originally  predicted,  adjusted 
overstrength  levels,  and  reduced  permanent  change  of  station  and 
subsistence  costs.  The  COST  portion  for  both  military  personnel  and 
operations  for  the  Army  decreased  slightly  from  the  initial  to  the  approved 
estimate.  Army  budget  officials  reported  that  the  approximately  $2  billion 
increase  in  operation  and  maintenance  costs  that  was  derived  outside  of 
COST  was  due  to  force  protection  and  other  equipment  or  services  needed 
to  support  the  troop  surge  and  the  additional  operations  performed  by 
surge  troops.  Table  4  provides  more  details  on  these  costs. 
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Table  4:  Non-COST  estimates  for  Army  Operation  and  Maintenance  Costs 

Dollars  in  millions 

Operation  and  Maintenance  Cost  Estimated 

Amount 

Estimated 

Force  protection 

$255 

Theater  maintenance  support 

763 

Communications  support 

198 

Predeployment  training 

447 

CONUS  based  support 

70 

Detainee  operations 

251 

Linguistics  and  intelligence  support 

200 

LOGCAP 

242 

Subsistence  transportation 

373 

Total  Non-COST  Estimate 

$2,798 

Source:  Army. 


Note:  Total  may  not  add  due  to  rounding. 

Similarly,  major  adjustments  were  made  to  the  non-COST  portions  of  the 
Air  Force,  Marine  Corps,  and  Navy  estimates  for  either  military  personnel 
costs,  operation  and  maintenance  costs,  or  both  appropriation  categories, 
depending  on  the  service.  These  changes  were  due  to  changed 
assumptions  regarding  deployment  and  specialized  needs.  For  example, 
the  Navy’s  non-COST  personnel  estimate  was  increased  for  costs 
associated  with  permanent  change  of  station  and  active  duty  for  special 
work  needs.  The  Marine  Corps  increased  its  non-COST  personnel  request 
by  $163  million  in  anticipation  of  increased  requirements  for  reserve  and 
Individual  Ready  Reserve  activations  to  active  duty.  The  adjustments  to 
the  Marine  Corps  and  Air  Force  COST-generated  portions  were  likewise 
substantial.  The  Marine  Corps’  increase  for  personnel  costs  reflected  in 
the  COST-attributed  portion  reflects  an  anticipated  increase  in  counter¬ 
insurgency  operations  in  Afghanistan,  while  the  Air  Force  estimate  was 
refined  to  exclude  a  KC-10  and  the  Air  National  Guard  and  Air  Force 
Reserve  personnel  that  would  accompany  the  KC-10.  From  this  case  study, 
it  is  clear  that  changing  the  assumptions  regarding  the  deployment  of 
personnel  and  equipment  can  have  substantial  impact  on  the  COST-related 
portion  of  a  GWOT  budget  request.  Also,  significant  changes  were  made  to 
the  non-COST  portion  of  the  request,  which  in  this  example  increased 
from  $775  million  to  about  $2.1  billion,  including  the  nearly  $1  billion 
offset  in  Army  personnel  cost  estimates.  The  information  regarding  the 
non-COST  portion  of  this  example  reveals  the  significance  and  size  of 
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estimates  generated  outside  the  tool,  even  in  a  situation  where  the 
majority,  if  not  all,  of  the  costs  would  be  assumed  to  be  related  to 
personnel  and  operations. 

We  did  not  validate  any  of  the  data  DOD  presented  in  the  above 
discussion,  or  any  of  the  assumptions  or  other  information  used  by  DOD 
to  develop  this  estimate. 
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UNDER  SECRETARY  OF  DEFENSE 
1  SCO  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-T  100 


Ms.  Sharon  Pickup 

Director,  Defense  Capabilities  and  Management 
D.S.  Govemment  Accountabilitj'  Office 
441  G  Street  N.W, 

Washington,  DC  20548 

Dear  Ms.  Pickup: 

Enclosed  is  the  Department  of  Defense  (DoD)  response  to  the  Government 
Accountability  Office  (GAO)  Draft  Report  GAO-08-982,  'DEFENSE  BUDGET: 
“Independent  Review  is  Needed  to  Ensure  DOD's  Use  of  Cost  Estimating  Tool  for 
Contingency  Operations  Follows  Best  Practices”  dated  July  28,  2008. 

The  Department  concurs  with  comment  on  both  of  the  recommendations  for 
executive  action  and  has  taken  action  to  address  tliem.  Thank  you  for  the  opportunity  to 
provide  the  Department’s  response  to  OAO’s  recommendations. 

Sincerely, 


y:,.;  y'-'i,'' 

Tina  W.  Jonas 

Enclosure: 

As  stated 
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Deparlinent  of  Defense  Comments 
GAO  CODE  35 1074/GAO-08-982 

“DEFENSE  BUDGET:  Independent  Review  is  Needed  to  Ensure  DoD’s  Use  of  Cost 
Estimating  Tool  for  Contingency  Operations  Follows  Best  Practices” 

department  of  defense  comments 

TO  THE  RECOMMENDATIONS 


DISCUSSION: 

•  'fhe  GAO  report  outlined  two  recommendations  for  executive  action. 

•  The  Department  of  Defense  concurs  with  comment  on  botli  items  for  executive  action. 

•  In  addition,  technical  comments  arc  provided  in  rcspimsc  to  the  draft  report. 

RECOMMENDATION  1:  The  Secretary  of  Defense  direct  the  USD(C)  to  “arrange  for  an 
independent  review  of  the  Contingency  Operations  Support  Tool  [COS'f]  against  best  practices 
for  cost  estimation.” 

DqD  comment  to  recommendation  1;  Concur  with  comment. 

•  I'he  Department  concurs  with  the  concept  of  an  independent  review  of  the  COST  model 
against  best  practices  for  cost  estimation.  In  fact,  the  Air  Force  Studies  and  Analyses 
Agency  (AFSAA)  conducted  such  a  review  in  2004  in  Iheir  report  titled  “Contingency 
Operations  Support  Tool  Ev'aluation:  The  Air  War  over  Serbia.”  The  report  includes  a 
comparison  of  the  model  output  to  actual  reported  costs  compiled  by  Dclcnsc  I'inancc 
and  Accoimting  Service.  Many  of  the  issues  that  AFSAA  articulated  within  the  report 
were  incorporated  by  the  Department  resulting  in  improvements  in  the  model. 

•  On  an  annual,  informal,  basis,  the  Services'  provide  recommendations  for  improvements 
to  the  model  that  would  better  meet  the  needs  of  each  individual  Service.  The  factors, 
processes,  and  algoritlims  are  updated  to  reflect  the  most  recent  assumptions  for  each 
Seiv'ice,  as  required.  Consistent  with  the  intetit  of  GAO’s  recommendation  and  effective 
immediately,  the  Under  Secrelarv'  of  Defense  (Comptroller)  has  issued  guidance  that  will 
be  incorporated  in  the  Financial  Management  Regulation.  ITie  guidance  includes  a 
process  for  updating  the  COS'f  model  to  ensure  it  reflects  the  most  current  bitdgetary 
assumptions  and  a  process  for  evaluating  the  functionality  of  the  model  to  determine  if 
adjustments  are  needed 

•  In  addition,  the  COST  model  is  updated  to  reflect  congressionally  determined  factors 
such  as  military  personnel  pay-rates  as  those  legislative  changes  become  law.  All  of 
these  procedures  bolster  the  continual  update  and  refinement  of  the  model. 

•  Finally  each  Service  uses  models  to  varying  degrees  to  create  and  review  cost  factors  for 
their  baseline  budgets.  The  COST  model  u.ses  many  of  these  same  factors.  The  Sciviccs 
have  procedures  in  place  to  validate  their  models,  'fhese  validated  factors  serve  as  the 
basis  for  the  Services'  baseline  budgets  and  are  the  foundation  upon  which  supplemental 
requirements  are  developed. 
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RECOMMENDATION  2:  The  Sccrctar>’  of  Defense  direct  the  USD(C)  to  “consider  options  for 
refining  COST,  determine  the  appropriate  items  or  types  ofeosts  for  which  COST  should  be 
applied,  and  identify  methods  to  be  used  when  COST  is  not  appropriate.” 

DoD  COMMENT  TO  RECOMMENDATION  2:  Concur  with  comment. 

•  Tlic  Department  uses  every  opportunity  to  refine  and  improve  the  COST  model  for 
camprchcnstvcncss  and  accuracy.  For  example,  during  a  supplemental  build  OSD 
Comptroller,  the  Joint  Staff,  Institute  for  Defense  Analyses  and  Service  personnel  review 
the  model  results  and  critique  assumptions,  factors,  and  outputs  of  the  COST  model.  The 
results  of  the  review  are  used  to  adjust  the  Supplemental  amounts  and  for  initiating 
improvements  to  the  COST  model. 

•  In  addition,  annual  training  involving  all  COST  users  is  conducted  before  .supplemental 
requests  are  developed.  The  feedback  and  exchange  between  the  attendees  {Services, 
OUSDfC).  the  Joint  Staff  and  on  occasion  the  Office  of  Management  and  Budget)  has 
resulted  in  changes  to  the  model  that  made  it  more  useful  and  accurate.  The  COST 
model  has  proven  itself  to  be  a  dynamic  and  adaptable  tool  that  is  used  successfully  by 
different  organizations  in  DoD  to  create  estimates  for  a  variety  of  force  deployments. 

•  As  for  supplemental  requirements  that  are  not  calculated  in  the  COST  model,  the 
Department  has  developed  an  extensive  review  process  for  each  non-COST  line  item. 
Each  Service  must  submit  all  non-COST  model  items  to  OUSD(C)  at  the  line  items  level 
of  detail  which  are  then  compared  to  historical  execution  and/or  anticipated  operational 
needs.  Only  those  line  items  that  are  fully  justified  are  included  in  the  supplemental 
request. 
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GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accoimtability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  pohcies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  rehability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
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A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
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more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  DC  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 
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